Systems and Methods of Determining Interaction Intent in Three-dimensional (3D) Sensory Space

ABSTRACT

The technology disclosed relates to determining intent for the interaction by calculating a center of effort for the applied forces. Movement of the points of virtual contacts and the center of effort are then monitored to determine a gesture-type intended for the interaction. The number of points of virtual contacts of the feeler zones and proximities between the points of virtual contacts are used to determine a degree of precision of a control object-gesture.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/851,058, entitled “SYSTEMS AND METHODS OF DETERMINING INTERACTIONINTENT IN THREE-DIMENSIONAL (3D) SENSORY SPACE,” filed Apr. 16, 2020(Attorney Docket No. ULTI 1030-4), which is a continuation of U.S.patent application Ser. No. 14/616,612, entitled “SYSTEMS AND METHODS OFDETERMINING INTERACTION INTENT IN THREE-DIMENSIONAL (3D) SENSORY SPACE,”filed Feb. 6, 2015 (Attorney Docket No. ULTI 1030-3), which claims thebenefit of U.S. provisional Patent Application No. 61/937,410, entitled,“SYSTEMS AND METHODS OF PROVIDING HAPTIC-LIKE FEEDBACK INTHREE-DIMENSIONAL (3D) SENSORY SPACE,” filed on Feb. 7, 2014 (AttorneyDocket No. LEAP 1030-1). The provisional and non-provisionalapplications are hereby incorporated by reference for all purposes.

This application is related to U.S. patent application Ser. No.14/616,603, entitled “SYSTEMS AND METHODS OF PROVIDING HAPTIC-LIKEFEEDBACK IN THREE-DIMENSIONAL (3D) SENSORY SPACE,” filed Feb. 6, 2015(Attorney Docket No. ULTI 1030-2). The related application isincorporated by reference for all purposes.

FIELD OF THE TECHNOLOGY DISCLOSED

The technology disclosed relates, in general, to interacting withvirtual objects in virtual spaces, and in particular implementations, toproviding haptic-like feedback for interactions and manipulations ofvirtual objects.

INCORPORATIONS

Materials incorporated by reference in this filing include thefollowing:

Velocity Field Interaction for Free Space Gesture Interface and Control,U.S. Prov. App. No. 61/891,880, filed 16 Oct. 2013 (Attorney Docket No.LEAP 1008-1/1009APR),

Interactive training recognition of free space gestures for interfaceand control, U.S. Prov. App. No. 61/872,538, filed 30 Aug. 2013(Attorney Docket No. LPM-013GPR),

Methods and systems for identifying position and shape of objects inthree-dimensional space, U.S. Prov. App. No. 61/587,554, filed 17 Jan.2012,

Systems and methods for capturing motion in three-dimensional space,U.S. Prov. App. No. 61/724,091, filed 8 Nov. 2012,

Non-tactile interface systems and methods, U.S. Prov. App. No.61/816,487, filed 26 Apr. 2013 (Attorney Docket No. LPM-028PR),

Dynamic user interactions for display control, U.S. Prov. App. No.61/752,725, filed 15 Jan. 2013,

Motion capture using cross-sections of an object, U.S. App. No.13/414,485, filed 7 Mar. 2012, and

System and methods for capturing motion in three-dimensional space, U.S.application Ser. No. 13/742,953, filed 16 Jan. 2013.

BACKGROUND

Manipulation of virtual objects is one of the most fundamentaloperations in virtual spaces. Virtual spaces are composed of interactivesimulations of real-world environments that provide users with a senseof being immersed in the interactive simulations. There is a perceivedneed to add haptic feedback to virtual spaces, which has led to a numberof prototype devices that simulate tactile sensations with applicationsof mechanical force, generally referred to as “haptic rendering.”

Most existing haptic rendering systems rely on specialized hardware wornby the subject while using the system and/or on the strategic placementof such hardware in the environment to provide the haptic rendering.Such systems tend to be expensive to construct. In addition, specializedhardware worn by the subject can be cumbersome and interfere with thesubject's natural movement. Further, systems involving large quantitiesof complex hardware tend not to operate in real time, due to the delaysinherent to mechanical devices. Such considerations of cost, complexityand convenience have limited the deployment and use of haptic renderingtechnology.

Also, the existing haptic rendering systems are limited with regards toproviding realistic responses of virtual objects to interactions. Forinstance, due to the inherent mechanical compliance of existing hapticinterface devices, the maximum rigidity of virtual objects is limitedsuch that a virtual object deforms to a greater degree in response to anapplied force than it would in real life. Thus, leading to a diminishedsense of realness to perception of virtual objects and degraded userexperience.

An opportunity arises to provide an economical approach that providesadvantages of haptic feedback for interaction with virtual objectswithout the draw backs of attaching or deploying specialized hardware.An opportunity also arises to eliminate unrealistic responses of virtualobjects to interactions and to avoid aberrations during manipulations ofthe virtual objects.

SUMMARY

The technology disclosed relates to providing sensory feedback for aninteraction between a control object in a three-dimensional (3D) sensoryspace and a virtual object in a virtual space that the control objectinteracts with. In some implementations, the 3D sensory space and thevirtual space can be overlapped, in part or in whole, to generate asynthetic or augmented space.

In particular, it relates to providing haptic-like and other sensoryfeedback to the user of a virtual space. Implementations includedefining a virtual feeler zone of the control object and generating fordisplay a feeler indicator that provides visual feedback over a range ofhover proximity of the feeler zone to the virtual object, applied forceon the virtual object resulting from virtual contact of the feeler zonewith the virtual object, and other material properties of the virtualobject.

The technology disclosed also relates to determining intent for aninteraction between a control object in a three-dimensional (3D) sensoryspace and a movable virtual object in a virtual space that the controlobject interacts with. In particular, it relates to defining a pluralityof feeler zones of the control object and calculating a center of effortfor the forces applied by the feeler zones on the virtual object.Movement of the points of virtual contacts and the center of effort arethen monitored to determine a gesture-type intended for the interaction.

The technology disclosed further relates to determining intent for aninteraction between a control object in a three-dimensional (3D) sensoryspace and a movable virtual object in a virtual space that the controlobject interacts with. In particular, it relates to defining a pluralityof feeler zones of the control object and calculating a center of effortfor the forces applied by the feeler zones on the virtual object. Thenumber of points of virtual contacts of the feeler zones and proximitiesbetween the points of virtual contacts are used to determine a degree ofprecision of the control object-gesture.

Other aspects and advantages of the technology disclosed can be seen onreview of the drawings, the detailed description and the claims, whichfollow.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to like partsthroughout the different views. Also, the drawings are not necessarilyto scale, with an emphasis instead generally being placed uponillustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings, inwhich:

FIG. 1A illustrates a system for capturing image data according to animplementation of the technology disclosed.

FIG. 1B is a simplified block diagram of a gesture-recognition systemimplementing an image analysis apparatus according to an implementationof the technology disclosed.

FIG. 2 shows one implementation of a feeler zone that reproducesmovements of a control object in a three-dimensional (3D) sensory space.

FIG. 3A illustrates one implementation of a feeler proximity-pressureindicator that provides visual feedback over a range of proximity andapplied force.

FIG. 3B is one implementation of modifying size of a feelerproximity-pressure indicator.

FIG. 3C illustrates one implementation of modifying shape of a feelerproximity-pressure indicator.

FIG. 3D shows one implementation of modifying opacity of a feelerproximity-pressure indicator.

FIG. 4A illustrates one implementation of a feeler object-densityindicator that provides visual feedback regarding density of a virtualobject.

FIG. 4B is one implementation of a feeler object-mass indicator thatprovides visual feedback regarding mass of a virtual object.

FIG. 4C depicts one implementation of a feeler surface-roughnessindicator that provides visual feedback regarding roughness of a virtualobject.

FIG. 4D illustrates one implementation of a feeler object-viscosityindicator that provides visual feedback regarding viscosity of a virtualobject.

FIG. 4E shows one implementation of a feeler surface-temperatureindicator that provides visual feedback regarding hotness or coldness ofa virtual object.

FIG. 5A is one implementation of manipulating a movable butnon-deformable virtual object.

FIG. 5B illustrates one implementation of manipulating an immovable butdeformable virtual object.

FIG. 5C shows one implementation of manipulating a movable anddeformable virtual object.

FIG. 6 depicts one implementation of generating for display a feelergrip-indicator that provides visual feedback regarding grip on a virtualobject.

FIG. 7A illustrates one implementation of identifying a intent asrotational.

FIG. 7B is one implementation of identifying a intent as translational.

FIG. 8A shows one implementation of determining a gesture to be a grossmotor gesture.

FIG. 8B is one implementation of determining a gesture to be a finemotor gesture.

FIG. 9 depicts a representative method of providing haptic-like feedbackfor an interaction between a control object in a three-dimensional (3D)sensory space and a virtual object in a virtual space that the controlobject interacts with.

FIG. 10 illustrates one implementation of a method of determining intentfor an interaction between a control object in a three-dimensional (3D)sensory space and a movable virtual object in a virtual space that thecontrol object interacts with.

FIG. 11 is a flowchart showing a method of determining a degree ofprecision of a gesture in accordance with implementations of thetechnology disclosed.

DESCRIPTION

Implementations of the technology disclosed relate to methods andsystems that provide sensory feedback to the users for theirinteractions with virtual objects in virtual spaces. This sensoryfeedback includes providing pseudo-haptic feedback, a haptic-likefeedback or perception of different physical properties of the virtualobjects using one or more of the human senses (e.g., visual or aural) ina virtualized world. In one implementation, haptic-like feedback orperception refers to the use of visual cues for providing informationabout material or physical properties of the virtual objects.

As used herein, a given signal, event or value is “dependent on” apredecessor signal, event or value of the predecessor signal, event orvalue influenced by the given signal, event or value. If there is anintervening processing element, step or time period, the given signal,event or value can still be “dependent on” the predecessor signal, eventor value. If the intervening processing element or step combines morethan one signal, event or value, the signal output of the processingelement or step is considered “dependent on” each of the signal, eventor value inputs. If the given signal, event or value is the same as thepredecessor signal, event or value, this is merely a degenerate case inwhich the given signal, event or value is still considered to be“dependent on” the predecessor signal, event or value. “Responsiveness”of a given signal, event or value upon another signal, event or value isdefined similarly.

Referring first to FIG. 1A, which illustrates an exemplarygesture-recognition system 100A including any number of cameras 102, 104coupled to a sensory-analysis system 106. Cameras 102, 104 can be anytype of camera, including cameras sensitive across the visible spectrumor, more typically, with enhanced sensitivity to a confined wavelengthband (e.g., the infrared (IR) or ultraviolet bands); more generally, theterm “camera” herein refers to any device (or combination of devices)capable of capturing an image of an object and representing that imagein the form of digital data. While illustrated using an example of a twocamera implementation, other implementations are readily achievableusing different numbers of cameras or non-camera light sensitive imagesensors or combinations thereof. For example, line sensors or linecameras rather than conventional devices that capture a two-dimensional(2D) image can be employed. The term “light” is used generally toconnote any electromagnetic radiation, which may or may not be withinthe visible spectrum, and may be broadband (e.g., white light) ornarrowband (e.g., a single wavelength or narrow band of wavelengths).

Cameras 102, 104 are preferably capable of capturing video images (i.e.,successive image frames at a constant rate of at least 15 frames persecond); although no particular frame rate is required. The capabilitiesof cameras 102, 104 are not critical to the technology disclosed, andthe cameras can vary as to frame rate, image resolution (e.g., pixelsper image), color or intensity resolution (e.g., number of bits ofintensity data per pixel), focal length of lenses, depth of field, etc.In general, for a particular application, any cameras capable offocusing on objects within a spatial volume of interest can be used. Forinstance, to capture motion of the hand of an otherwise stationaryperson, the volume of interest can be defined as a cube approximatelyone meter on a side.

In some implementations, the illustrated system 100A includes one ormore sources 108, 110, which can be disposed to either side of cameras102, 104, and are controlled by sensory-analysis system 106. In oneimplementation, the sources 108, 110 are light sources. For example, thelight sources can be infrared light sources, e.g., infraredlight-emitting diodes (LEDs), and cameras 102, 104 can be sensitive toinfrared light. Use of infrared light can allow the gesture-recognitionsystem 100A to operate under a broad range of lighting conditions andcan avoid various inconveniences or distractions that may be associatedwith directing visible light into the region where the person is moving.However, a particular wavelength or region of the electromagneticspectrum can be required. In one implementation, filters 120, 122 areplaced in front of cameras 102, 104 to filter out visible light so thatonly infrared light is registered in the images captured by cameras 102,104. In another implementation, the sources 108, 110 are sonic sourcesproviding sonic energy appropriate to one or more sonic sensors (notshown in FIG. 1A for clarity sake) used in conjunction with, or insteadof, cameras 102, 104. The sonic sources transmit sound waves to theuser; the user either blocks (or “sonic shadowing”) or alters the soundwaves (or “sonic deflections”) that impinge upon her. Such sonic shadowsand/or deflections can also be used to detect the user's gestures and/orprovide presence information and/or distance information using rangingtechniques known in the art. In some implementations, the sound wavesare, for example, ultrasound, that is not audible to humans.

It should be stressed that the arrangement shown in FIG. 1A isrepresentative and not limiting. For example, lasers or other lightsources can be used instead of LEDs. In implementations that includelaser(s), additional optics (e.g., a lens or diffuser) may be employedto widen the laser beam (and make its field of view similar to that ofthe cameras). Useful arrangements can also include short- and wide-angleilluminators for different ranges. Light sources are typically diffuserather than specular point sources; for example, packaged LEDs withlight-spreading encapsulation are suitable.

In operation, light sources 108, 110 are arranged to illuminate a regionof interest 112 that includes a control object portion 114 (in thisexample, a hand) that may optionally hold a tool or other object ofinterest and cameras 102, 104 are oriented toward the region 112 tocapture video images of the hand 114. In some implementations, theoperation of light sources 108, 110 and cameras 102, 104 is controlledby the sensory-analysis system 106, which can be, e.g., a computersystem, control logic implemented in hardware and/or software orcombinations thereof. Based on the captured images, sensory-analysissystem 106 determines the position and/or motion of object 114.

FIG. 1B is a simplified block diagram of a computer system 100B,implementing sensory-analysis system 106 (also referred to as an imageanalyzer) according to an implementation of the technology disclosed.Sensory-analysis system 106 can include or consist of any device ordevice component that is capable of capturing and processing image data.In some implementations, computer system 100B includes a processor 132,memory 134, a sensor interface 136, a display 138 (or other presentationmechanism(s), e.g. holographic projection systems, wearable googles orother head mounted displays (HMDs), heads up displays (HUDs), othervisual presentation mechanisms or combinations thereof, speakers 139, akeyboard 140, and a mouse 141. Memory 134 can be used to storeinstructions to be executed by processor 132 as well as input and/oroutput data associated with execution of the instructions. Inparticular, memory 134 contains instructions, conceptually illustratedas a group of modules described in greater detail below, that controlthe operation of processor 132 and its interaction with the otherhardware components. An operating system directs the execution oflow-level, basic system functions such as memory allocation, filemanagement and operation of mass storage devices. The operating systemmay be or include a variety of operating systems such as MicrosoftWINDOWS operating system, the Unix operating system, the Linux operatingsystem, the Xenix operating system, the IBM AIX operating system, theHewlett Packard UX operating system, the Novell NETWARE operatingsystem, the Sun Microsystems SOLARIS operating system, the OS/2operating system, the BeOS operating system, the MAC OS operatingsystem, the APACHE operating system, an OPENACTION operating system,iOS, Android or other mobile operating systems, or another operatingsystem platform.

The computing environment can also include otherremovable/non-removable, volatile/nonvolatile computer storage media.For example, a hard disk drive can read or write to non-removable,nonvolatile magnetic media. A magnetic disk drive can read from or writeto a removable, nonvolatile magnetic disk, and an optical disk drive canread from or write to a removable, nonvolatile optical disk such as aCD-ROM or other optical media. Other removable/non-removable,volatile/nonvolatile computer storage media that can be used in theexemplary operating environment include, but are not limited to,magnetic tape cassettes, flash memory cards, digital versatile disks,digital video tape, solid state RAM, solid state ROM, and the like. Thestorage media are typically connected to the system bus through aremovable or non-removable memory interface.

Processor 132 can be a general-purpose microprocessor, but depending onimplementation can alternatively be a microcontroller, peripheralintegrated circuit element, a CSIC (customer-specific integratedcircuit), an ASIC (application-specific integrated circuit), a logiccircuit, a digital signal processor, a programmable logic device such asan FPGA (field-programmable gate array), a PLD (programmable logicdevice), a PLA (programmable logic array), an RFID processor, smartchip, or any other device or arrangement of devices that is capable ofimplementing the actions of the processes of the technology disclosed.

Sensor interface 136 can include hardware and/or software that enablescommunication between computer system 100B and cameras such as cameras102, 104 shown in FIG. 1A, as well as associated light sources such aslight sources 108, 110 of FIG. 1A. Thus, for example, sensor interface136 can include one or more data ports 146, 148 to which cameras can beconnected, as well as hardware and/or software signal processors tomodify data signals received from the cameras (e.g., to reduce noise orreformat data) prior to providing the signals as inputs to amotion-capture (“mocap”) program 144 executing on processor 132. In someimplementations, sensor interface 136 can also transmit signals to thecameras, e.g., to activate or deactivate the cameras, to control camerasettings (frame rate, image quality, sensitivity, etc.), or the like.Such signals can be transmitted, e.g., in response to control signalsfrom processor 132, which can in turn be generated in response to userinput or other detected events.

Sensor interface 136 can also include controllers 147, 149, to whichlight sources (e.g., light sources 108, 110) can be connected. In someimplementations, controllers 147, 149 provide operating current to thelight sources, e.g., in response to instructions from processor 132executing mocap program 144. In other implementations, the light sourcescan draw operating current from an external power supply, andcontrollers 147, 149 can generate control signals for the light sources,e.g., instructing the light sources to be turned on or off or changingthe brightness. In some implementations, a single controller can be usedto control multiple light sources.

Instructions defining mocap program 144 are stored in memory 134, andthese instructions, when executed, perform motion-capture analysis onimages supplied from cameras connected to sensor interface 136. In oneimplementation, mocap program 144 includes various modules, such as anobject detection module 152, an object analysis module 154, and agesture-recognition module 156. Object detection module 152 can analyzeimages (e.g., images captured via sensor interface 136) to detect edgesof an object therein and/or other information about the object'slocation. Object analysis module 154 can analyze the object informationprovided by object detection module 152 to determine the 3D positionand/or motion of the object (e.g., a user's hand). Examples ofoperations that can be implemented in code modules of mocap program 144are described below. Memory 134 can also include other informationand/or code modules used by mocap program 144 such as an applicationplatform 166 that allows a user to interact with the mocap program 144using different applications like application 1 (App1), application 2(App2), and application N (AppN).

Display 138, speakers 139, keyboard 140, and mouse 141 can be used tofacilitate user interaction with computer system 100B. In someimplementations, results of gesture capture using sensor interface 136and mocap program 144 can be interpreted as user input. For example, auser can perform hand gestures that are analyzed using mocap program144, and the results of this analysis can be interpreted as aninstruction to some other program executing on processor 132 (e.g., aweb browser, word processor, or other application). Thus, by way ofillustration, a user might use upward or downward swiping gestures to“scroll” a webpage currently displayed on display 138, to use rotatinggestures to increase or decrease the volume of audio output fromspeakers 139, and so on.

It will be appreciated that computer system 100B is illustrative andthat variations and modifications are possible. Computer systems can beimplemented in a variety of form factors, including server systems,desktop systems, laptop systems, tablets, smart phones or personaldigital assistants, wearable devices, e.g., goggles, head mounteddisplays (HMDs), wrist computers, and so on. A particular implementationcan include other functionality not described herein, e.g., wired and/orwireless network interfaces, media playing and/or recording capability,etc. In some implementations, one or more cameras can be built into thecomputer or other device into which the sensor is imbedded rather thanbeing supplied as separate components. Further, an image analyzer can beimplemented using only a subset of computer system components (e.g., asa processor executing program code, an ASIC, or a fixed-function digitalsignal processor, with suitable I/O interfaces to receive image data andoutput analysis results).

While computer system 100B is described herein with reference toparticular blocks, it is to be understood that the blocks are definedfor convenience of description and are not intended to imply aparticular physical arrangement of component parts. Further, the blocksneed not correspond to physically distinct components. To the extentthat physically distinct components are used, connections betweencomponents (e.g., for data communication) can be wired and/or wirelessas desired.

With reference to FIGS. 1A and 1B, the user performs a gesture that iscaptured by the cameras 102, 104 as a series of temporally sequentialimages. In other implementations, cameras 102, 104 can capture anyobservable pose or portion of a user. For instance, if a user walks intothe field of view near the cameras 102, 104, cameras 102, 104 cancapture not only the whole body of the user, but the positions of armsand legs relative to the person's core or trunk. These are analyzed by agesture-recognition module 156, which can be implemented as anothermodule of the mocap 144. Gesture-recognition module 156 provides inputto an electronic device, allowing a user to remotely control theelectronic device and/or manipulate virtual objects, such asprototypes/models, blocks, spheres, or other shapes, buttons, levers, orother controls, in a virtual environment displayed on display 138. Theuser can perform the gesture using any part of her body, such as afinger, a hand, or an arm. As part of gesture recognition orindependently, the sensory-analysis system 106 can determine the shapesand positions of the user's hand in 3D space and in real time; see,e.g., U.S. Ser. Nos. 61/587,554, 13/414,485, 61/724,091, and 13/724,357filed on Jan. 17, 2012, Mar. 7, 2012, Nov. 8, 2012, and Dec. 21, 2012respectively, the entire disclosures of which are hereby incorporated byreference. As a result, the sensory-analysis system 106 can not onlyrecognize gestures for purposes of providing input to the electronicdevice, but can also capture the position and shape of the user's handin consecutive video images in order to characterize the hand gesture in3D space and reproduce it on the display screen 138.

In one implementation, the gesture-recognition module 156 compares thedetected gesture to a library of gestures electronically stored asrecords in a database, which is implemented in the sensory-analysissystem 106, the electronic device, or on an external storage system. (Asused herein, the term “electronically stored” includes storage involatile or non-volatile storage, the latter including disks, Flashmemory, etc., and extends to any computationally addressable storagemedia (including, for example, optical storage).) For example, gesturescan be stored as vectors, i.e., mathematically specified spatialtrajectories, and the gesture record can have a field specifying therelevant part of the user's body making the gesture; thus, similartrajectories executed by a user's hand and head can be stored in thedatabase as different gestures so that an application can interpret themdifferently.

Feeler Zone and Feeler Indicator

FIG. 2 shows one implementation of a feeler zone 200 that reproducesmovements a control object in a three-dimensional (3D) sensory space. Invarious implementations, after a user successfully initiatescommunications with the gesture-recognition system 156, thegesture-recognition system 156 generates a feeler zone 200 or a FIG. 200(hereafter “feeler zone”) representing a detected control object, inthis case, a body's part (e.g., a hand) and displays it across thedisplay 138. In one implementation, the gesture-recognition system 156coherently locks the movement of the feeler zone 200 on the display 138to follow the actual motion of the user's gesture. For example, when theuser moves her hand in the upward direction, the displayed feeler zone200 also moves upward on the display screen in response.

As a result, the motion of the feeler zone 200 directly maps usergestures to displayed content such that, for example, the user's handand the feeler zone 200 behave like a PC mouse and a cursor on themonitor, respectively. This allows the user to evaluate the relationshipbetween actual physical gesture movement and the resulting actionstaking place on the display 138, e.g., movement of virtual objectsdisplayed thereon. The absolute position of the hand, therefore, istypically not important for display control; rather, a relative positionand/or direction of movement of the user's body controls the on-screenaction, e.g., the movement of feeler zone 200.

In some implementations, feeler zone 200 can include sub-components thattrack portions of the control object, such as fingers of a hand. Asshown in FIG. 2 , a particular feeler zone 212 reproduces movements of aleft index finger. In one implementation, feeler zone 212 is a virtualextension of the control object that is not one-to-one mapped to thecontrol object. In another implementation, feeler zone 212 is adeformable region of the control object that proportionally depressesresponsive to virtual contact with a virtual contact and pressureapplied by the control object.

Referring to feeler indicator 202, it provides visual feedback regardinghaptic properties (e.g., properties perceptible by pain (e.g.,sharpness, pointedness), pressure (e.g., hardness, rigidity,flexibility), temperature, (e.g., hot, cold), somatosensory perceptionof surface characteristics (e.g., edges, curvature, texture, patternrecognition, smoothness, coarseness), other sensations perceptible bytouch alone, or in conjunction with one or more other senses such asvision, hearing, smell, taste, proprioception, equilibrioception, andnociception, and combinations thereof) of virtual objects in virtualspace. In one implementation, feeler indicator 202 is defined forparticular portions of the control object like fingers and located atdistal ends of the feeler zone 212, as shown in FIG. 2 . Feelerindicator 202 can be receptive of different haptic properties of thevirtual objects, including density, mass, viscosity, surface texture, ortemperature. In some implementations, custom logic can be defined forfeeler indicator 202 such that a physical attribute of the feelerindicator 202 like color, shape, size, or opacity can correspond todifferent threshold magnitudes of different haptic properties of thevirtual objects. For instance, color blue can represent that aparticular virtual object is less than two pounds or kilograms per cubicmeter, color orange can indicate that the virtual object is between twoand four pounds or kilograms per cubic meter, and color red canrepresent that the virtual object is more than four pounds or kilogramsper cubic meter. In other implementations, physical attributes of thefeeler indicator 202 are linked to proximity of the feeler zone 212 to avirtual object. In one example, as the feeler zone 212 approaches avirtual object; a color, shape, size, or opacity of the feeler indicator202, or the combination thereof, is modified.

FIG. 3A illustrates one implementation of a feeler proximity-pressureindicator 300A that provides visual feedback over a range of proximityand applied force. At action 300A1, feeler zone 212 is at an initialdistance from the virtual object 302. In one implementation, customlogic for feeler zone 212 is defined such that particular feelerindicators that are closer to the virtual object 302 are darker than theones that are further away. As shown in FIG. 3A, feelerproximity-pressure indicator 202C is darker than the feelerproximity-pressure indicator 202A because the feeler zone 212C (middlefinger) is closer to the virtual object 302 than the feeler zone 212A(thumb).

At action 300A2, as the feeler zone 212 approaches the virtual object302 in response to movement of the control object in the 3D space,particular feeler proximity-pressure indicators 202A-E continue tobecome darker proportional to their respective hover proximities to thevirtual object 302. Further, at action 300A3, when the feeler zone 212approaches the virtual object 302 within an initial hover proximitythreshold, the feeler proximity-pressure indicator 202 snaps to thesurface of the virtual object 302. As a result, the particular feelerproximity-pressure indicators 202B-D that touch the virtual object 302have darker color densities than the ones that do not touch the virtualobject 302 i.e. feeler proximity-pressure indicators 202A and 202E. Inone implementation, proximity between the feeler zone 212 and virtualobject 302 directly or inversely proportionates a physical attribute ofthe feeler proximity-pressure indicator 202 such as its color, shape,size, or opacity, as discussed later in this application.

Subsequent to the virtual contact at action 300A3, feeler zone 212applies pressure on the virtual object 302. In one implementation, aspring function calculates the pressure applied by the virtual contactover a compressible range. For instance, the pressure is calculatedbased on Hooke's law (F=−kΔx) such that the amount of force applied isinversely proportional to the compression of the feeler zone 212. Inanother implementation, the magnitude of force applied directly orinversely proportionates a physical attribute of the feelerproximity-pressure indicator 202 such as its color, shape, size, oropacity, as discussed below.

FIG. 3B-D show different implementations of modifying size, shape, andopacity of the feeler proximity-pressure indicator 202 over a range ofproximity with virtual object 302 and pressure applied by the feelerzone 212 on the virtual object 302. As FIG. 3B depicts, size of thefeeler proximity-pressure indicator 202 increases as the feeler zone 212moves closer to the virtual object 302 from action 300B1 to action300B2. After coming in contact with the virtual object at action 300B2,size of the feeler proximity-pressure indicator 202 continues toincrease responsive to increase in the pressure applied by the feelerzone 212.

In another implementation, as shown in FIG. 3C, as the feeler zone 212approaches the virtual object 302 and further applies pressure on itfrom action 300C1 to action 300C3, shape of the feelerproximity-pressure indicator 202 changes from a quadrilateral, pentagon,and hexagon respectively. In yet another implementation, as shown inFIG. 3D, a change in opacity of the feeler proximity-pressure indicator202 occurs as a function of distance between the feeler zone 212 andvirtual object 302 and pressure applied by the feeler zone 212 on thevirtual object 302, making the feeler proximity-pressure indicator 202transparent, translucent, and opaque at actions 300C1-C3 respectively.

Haptic-Like Feedback

Some implementations include assigning haptic, material, or physicalproperties like mass, density, viscosity, elasticity, texture, ortemperature to virtual objects. The haptic, material, or physicalproperties can be obtained from reference sources or experimentallymeasured data. In one implementation, these properties can be linear ornon-linear, isotropic, or anisotropic, homogeneous or inhomogeneous. Thegesture-recognition system 156 then generates virtual objects thatsimulate these properties.

FIG. 4A illustrates one implementation of a feeler object-densityindicator 400A that provides visual feedback regarding density of avirtual object. In one implementation, the density value of a virtualobject is dependent on the volume and pixel density of a voxel grid. Inanother implementation, the density value is designated. According tosome implementations, one or more physical attributes of the feelerobject-density indicator can be a function of the density values of thevirtual objects such that color, shape, size, or opacity of the feelerobject-density indicator are proportionally modified responsive todensity values of the virtual objects that the feeler object-densityindicator comes in contact with.

In other implementations, density values for high, medium, and lowdensity thresholds can be set and corresponding physical attributes ofthe feeler object-density indicator can be associated with thesethresholds. For instance, as shown in FIG. 4A, particular feelerobject-density indicator 202C has a lower color density when it comes invirtual contact with a low-density virtual object 402 at action 400A1and a higher color density when it comes in virtual contact with ahigh-density virtual object 412 at action 400A2.

In yet another implementation, visual feedback from the feelerobject-density indicator can be calibrated based on densities of virtualobjects that it comes in contact with. For instance, custom logic forfeeler object-density indicator can be defined such that a particularvalue of a physical attribute of the feeler object-density indicator ismapped to an “optimal gestural effort” or “optimal gesturalcompleteness” required to manipulate the virtual object proportional todensity of the virtual object. In one example, color green can indicatethat the applied effort, pressure, or force is adequate to displace avirtual object with particular density value. Also, color red canindicate that applied effort, pressure, or force is more than what isoptimally required and can result in deforming the virtual object.Further, in some implementations, if the applied effort, pressure, orforce is less than what is required to displace or deform the virtualobject or the virtual object is immovable or non-deformable, but thecontrol object continues its motion in the 3D sensory space, the feelerzone and feeler object-density indicator are automatically constrainedand prevented from displacing or penetrating the virtual object.

FIG. 4B is one implementation of a feeler object-mass indicator 400Bthat provides visual feedback regarding mass of a virtual object. In oneimplementation, the mass of a virtual object is dependent on the pixelarea covered by the virtual object across a display. In anotherimplementation, the mass is designated. According to someimplementations, one or more physical attributes of the feelerobject-mass indicator can be a function of the masses of the virtualobjects such that color, shape, size, or opacity of the feelerobject-mass indicator are proportionally modified responsive to massvalues of the virtual objects that the feeler object-mass indicatorcomes in contact with.

In other implementations, values for high, medium, and low massthresholds can be set and corresponding physical attributes of thefeeler object-mass indicator can be associated with these thresholds.For instance, as shown in FIG. 4B, particular feeler object-densityindicator 202C has a lower color density when it comes in virtualcontact with a low-mass virtual object 422 at action 400B1 and a highercolor density when it comes in virtual contact with a high-mass virtualobject 432 at action 400B2.

In yet another implementation, visual feedback from the feelerobject-mass indicator can be calibrated based on masses of virtualobjects that it comes in contact with. For instance, custom logic forfeeler object-mass indicator can be defined such that a particular valueof a physical attribute of the feeler object-mass indicator is mapped toan “optimal gestural effort” or “optimal gestural completeness” requiredto manipulate the virtual object proportional to mass of the virtualobject. In one example, color green can indicate that the appliedeffort, pressure, or force is adequate to displace a virtual object withparticular mass. Also, color red can indicate that applied effort,pressure, or force is more than what is optimally required and canresult in deforming the virtual object. Further, in someimplementations, if the applied effort, pressure, or force is less thanwhat is required to displace or deform the virtual object or the virtualobject is immovable or non-deformable, but the control object continuesits motion in the 3D sensory space, the feeler zone and feelerobject-density indicator are automatically constrained and preventedfrom displacing or penetrating the virtual object.

FIG. 4C depicts one implementation of a feeler surface-roughnessindicator 400C that provides visual feedback regarding roughness of avirtual object. In one implementation, roughness magnitude of thevirtual object is a function of a coefficient of friction. Thecoefficient of friction is based on virtual textures or patterns andtheir grooving, inter-element spacing, or element size. In anotherimplementation, the coefficient of friction is designated. As shown inFIG. 4C, particular feeler object-density indicator 202C has a lowercolor density when it comes in virtual contact with a cloth-like surfaceof virtual object 442 at action 400C1 and a higher color density when itcomes in virtual contact with a sandpaper-like surface of virtual object452 at action 400C2.

FIG. 4D illustrates one implementation of a feeler object-viscosityindicator 400D that provides visual feedback regarding viscosity of avirtual object. In one implementation, viscosity of the virtual objectis based on Young's modulus of viscoelasticity. As shown in FIG. 4D,particular feeler object-density indicator 202C has a lower colordensity when it comes in virtual contact with a low-viscous virtualobject 462 at action 400D1 and a higher color density when it comes invirtual contact with a high-viscous virtual object 472 at action 400D2.

FIG. 4E shows one implementation of a feeler surface-temperatureindicator 400E that provides visual feedback regarding hotness orcoldness of a virtual object. In one implementation, temperature of thevirtual object is designated. In another implementation, object color isused as a proxy for object temperature. As shown in FIG. 4E, particularfeeler object-density indicator 202C has a lower color density when itcomes in virtual contact with a 35° F. virtual object 482 at action400E1 and a higher color density when it comes in virtual contact with a60° F. virtual object 482 at action 400E2.

Haptic-Like Manipulation

FIG. 5A is manipulation 500A of a movable but non-deformable virtualobject 512, according to one implementation. At action 500A1, as thefeeler zone 212 approaches the virtual object 512 in response tomovement of the control object in the 3D space, particular feelerproximity-pressure indicators 202A-E continue to become darkerproportional to their respective hover proximities to the virtual object512. Further, at action 500A2, when the feeler zone 212 approaches thevirtual object 512 within an initial hover proximity threshold, thefeeler proximity-pressure indicator 202 snaps to the surface of thevirtual object 512. Then, at action 500A3, in response to virtualcontact of the feeler proximity-pressure indicator 202 with the virtualobject 512, virtual object 512 is displaced by distance x. As a result,displacement of the virtual object 512 is depicted in the generateddisplay (512A) without depicting penetration of the virtual object 512by the feeler zone 212, while providing the visual feedback regardingthe pressure applied by the virtual contact. Finally, the calculatedpressure applied by the virtual contact is provided as input togesture-recognition module 156 for further manipulating the virtualobject 512.

FIG. 5B illustrates manipulation 500B of an immovable but deformablevirtual object 522, according to one implementation. At action 500B1, asthe feeler zone 212 approaches the virtual object 522 in response tomovement of the control object in the 3D space, particular feelerproximity-pressure indicators 202A-E continue to become darkerproportional to their respective hover proximities to the virtual object522. Further, at action 500B2, when the feeler zone 212 approaches thevirtual object 522 within an initial hover proximity threshold, thefeeler proximity-pressure indicator 202 snaps to the surface of thevirtual object 522. Then, at action 500B3, in response to a period ofvirtual contact of the feeler proximity-pressure indicator 202 with thevirtual object 522, virtual object 522 is deformed by depth d. As aresult, deformation of the virtual object 522 is depicted (522A) in thegenerated display without depicting penetration of the virtual object522 by the feeler zone 202, while providing the visual feedbackregarding the pressure applied by the virtual contact. Finally, thecalculated pressure applied by the virtual contact is provided as inputto gesture-recognition module 156 for further manipulation of thevirtual object 522.

FIG. 5C illustrates manipulation 500C of a movable and deformablevirtual object 532, according to one implementation. At action 500C1, asthe feeler zone 212 approaches the virtual object 532 in response tomovement of the control object in the 3D space, particular feelerproximity-pressure indicators 202A-E continue to become darkerproportional to their respective hover proximities to the virtual object532. Further, at action 500C2, when the feeler zone 212 approaches thevirtual object 532 within an initial hover proximity threshold, thefeeler proximity-pressure indicator 202 snaps to the surface of thevirtual object 532. Then, at action 500C3, in response to a period ofvirtual contact of the feeler proximity-pressure indicator 202 with thevirtual object 532, virtual object 532 is displaced by distance x anddeformed by depth d. As a result, displacement and deformation of thevirtual object 532 is depicted (532A) in the generated display withoutdepicting penetration of the virtual object 532 by the feeler zone 202,while providing the visual feedback regarding the pressure applied bythe virtual contact. Finally, the calculated pressure applied by thevirtual contact is provided as input to gesture-recognition module 156for further manipulation of the virtual object 532.

Some implementations include depicting, in the generated display,inertia and non-deformability of the virtual object without depictingpenetration of the virtual object by the feeler zone while providing thevisual feedback regarding the pressure applied by the virtual contactand providing calculated pressure applied by the virtual contact asinput to an object movement system. In one implementation,non-deformability refers to an initial form, shape, or size of a virtualobject being preserved or maintained after a contact event. Thecalculated pressure applied by the virtual contact is then provided asinput to gesture-recognition module 156 for further manipulation of thevirtual object 532.

Other implementations include a feeler proximity-pressure audiogenerator that provides audio feedback regarding proximity of the feelerzone to the virtual object and escalation from proximity to pressureapplied by virtual contact of the feeler zone with the virtual object.In one implementation, sound waves of varying frequencies and pitchesare used to indicate different threshold magnitudes of different hapticproperties of the virtual objects. In some other implementations, feeleraudio generators can be used to provide audio feedback regarding otherhaptic properties of a virtual object such as density, surface texture,mass, viscosity, plasticity, elasticity, or temperature. In yet otherimplementations, feeler vibrators can be used to provide tactilefeedback regarding the different haptic properties through vibrationfrequencies of varying magnitudes.

Virtual Grip

FIG. 6 depicts one implementation of generating for display a feelergrip-indicator 600 that provides visual feedback regarding grip on avirtual object 622. At action 600A1, when feeler zones 612A-E approachthe virtual object 622 within an initial hover proximity threshold, thefeeler grip-indicator 600 grasps the virtual object 622. After graspingthe virtual object 622 at action 600A2, feeler zones 612A-E, applyopposing forces on the virtual object 622. In some implementations,feeler-grip indicators 602A-E provide visual feedback regardingdifferent magnitudes of pressure applied by feeler zones 612A-Ecorresponding to different portions of a control object (e.g. fingers ofa hand).

In some implementations, custom logic for feeler-grip indicators 602A-Ecan be defined such that a particular value of a physical attribute ofthe feeler object-mass indicator is mapped to an “optimal gesturaleffort” or “optimal gestural completeness” required to manipulate thevirtual object 622 proportional to mass, density, surface texture,elasticity, plasticity, or viscosity of the virtual object 622. In oneexample, color green can indicate that the applied effort, pressure, orforce is adequate for creating a strong grip and to lift a virtualobject with particular mass, density value, surface texture-type,elasticity level, plasticity level, or viscosity level. Also, color redcan indicate that applied effort, pressure, or force is more than whatis optimally required and can result in deforming the virtual object622. Further, in some implementations, if the applied effort, pressure,or force is less than what is required to properly grasp or lift thevirtual object 622 is immovable or non-graspable, but the control objectcontinues its motion in the 3D sensory space, the feeler zones 612A-Eand feeler-grip indicators 602A-E are automatically constrained andprevented from displacing or penetrating the virtual object 622.

Intent

FIG. 7A illustrates identification 700A of a intent as rotational,according to one implementation. At action 700A1, a plurality of forcesare applied to a movable virtual object 705 at one or more points ofvirtual contacts 702A-D of feeler zones 202A-D with the virtual object705. At action 700A2, a center of effort 712 is calculated for theforces applied by the feeler zones 202A-D. In one implementation, thecenter of effort 712 is calculated by determining a geometric centroidof the polygon formed by the applied forces. For further informationregarding calculating geometric centroids, reference can be made toWolfram MathWorld, athttp://mathworld.wolfram.com/GeometricCentroid.html, Geometric Centroid,on Feb. 4, 204, 03:21 UTC.

In some implementations, a gesture-type is determined based on movementof the points of virtual contacts 702A-D with respect to the center ofeffort 712. At action 700A3, the gesture-type is identified as arotational stroke in response to detecting rotation of the points ofvirtual contacts around the center of effort 712. The virtual object 705is then rotated at action 700A3 in response to the rotational stroke.

FIG. 7B shows identification 700B of a intent as translational,according to one implementation. At action 700B1, a plurality of forcesare applied to a movable virtual object 705 at one or more points ofvirtual contacts 722A-D of feeler zones 202A-D with the virtual object705. At action 700B2, a center of effort 732 is calculated for theforces applied by the feeler zones 202A-D. In one implementation, thecenter of effort 732 is calculated by determining a geometric centroidof the polygon formed by the applied forces. For further informationregarding calculating geometric centroids, reference can be made toWolfram MathWorld, athttp://mathworld.wolfram.com/GeometricCentroid.html, Geometric Centroid,on Feb. 4, 204, 03:21 UTC.

In some implementations, a gesture-type is determined based on movementof the center of effort 732. At action 700B3, the gesture-type isidentified as a translational stroke in response to detectingdisplacement of the center of effort 732. The virtual object 705 is thentranslated at action 700B3 in response to the translational stroke.

Gestural Precision

FIG. 8A shows determination 800A of a gesture as a gross motor gesture,according to one implementation. At action 800A1, a plurality of forcesare applied to a movable virtual object 805, responsive to a controlobject-gesture, at one or more points of virtual contacts 812A-D offeeler zones 202A-D with the virtual object 805. In someimplementations, a degree of precision of the control object-gesture isdetermined dependent on a number of points of virtual contacts 812A-D ofthe feeler zones 202A-B and proximities between the points of virtualcontacts 812A-D. At action 800A2, the control object-gesture isdetermined to be a gross motor gesture when the degree of precisiondeceeds a precision threshold value. In one implementation, theprecision threshold value can be based on an absolute number, apercentage, or a distance measurement.

FIG. 8B shows determination 800B of a gesture as a fine motor gesture,according to one implementation. At action 800B1, a plurality of forcesare applied to a movable virtual object 805, responsive to a controlobject-gesture, at one or more points of virtual contacts 822A-B offeeler zones 202A-B with the virtual object 805. In someimplementations, a degree of precision of the control object-gesture isdetermined dependent on a number of points of virtual contacts 822A-B ofthe feeler zones 202A-D and proximities between the points of virtualcontacts 822A-B. At action 800B2, the control object-gesture isdetermined to be a fine motor gesture when the degree of precisionexceeds a precision threshold value. In one implementation, theprecision threshold value can be based on an absolute number, apercentage, or a distance measurement.

Flowcharts

FIG. 9 depicts a representative method 900 of providing haptic feedbackfor an interaction between a control object in a three-dimensional (3D)sensory space and a virtual object in a virtual space that the controlobject interacts with. At action 902, a virtual feeler zone of thecontrol object is defined. In one implementation, the virtual feelerzone reproduces movements of the control object in a virtual space thatare captured in the 3D sensory space. In another implementation, asynthetic or augmented space is created by overlaying the virtual spaceon a physical space and the virtual feeler zone reproduces movements ofthe control object in the synthetic or augmented space.

At action 912, a feeler proximity-pressure indicator is generated fordisplay that provides visual feedback regarding proximity of the feelerzone to the virtual object and escalation from proximity to pressureapplied by virtual contact of the feeler zone with the virtual object.At action 922, the feeler proximity-pressure indicator is snapped to asurface of the virtual object when the feeler zone approaches thevirtual object within an initial hover proximity threshold.

At action 932, appearance of the feeler proximity-pressure indicator ismodified responsive to distance between the feeler zone and the virtualobject. In one implementation, size of the feeler proximity-pressureindicator is modified responsive to distance between the feeler zone andthe virtual object. In another implementation, shape of the feelerproximity-pressure indicator is modified responsive to distance betweenthe feeler zone and the virtual object. In yet another implementation,opacity of the feeler proximity-pressure indicator is modifiedresponsive to distance between the feeler zone and the virtual object.

At action 942, a speed of approach of the feeler zone of the controlobject to a surface of the virtual object is tracked and an initialhover proximity threshold and calibration of the visual feedback isadjusted responsive to the speed of approach. In one implementation, theinitial hover proximity threshold distance is adjusted proportionally tothe speed of approach.

At action 952, opposing forces applied by a plurality of feeler zones toa movable virtual object are calculated and a feeler grip-indicator isdisplayed that provides visual feedback regarding whether the appliedopposing forces are sufficient to grasp the virtual object proportionalto the mass of the virtual object.

At action 962, the virtual object is manipulated responsive to thepressure applied by the virtual contact. In one implementation, themanipulation includes shaking, moving, or rotating the virtual object.In another implementation, manipulation includes selecting, grabbing, orlifting the virtual object.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. In the interest ofconciseness, the combinations of features disclosed in this applicationare not individually enumerated and are not repeated with each base setof features. The reader will understand how features identified in thissection can readily be combined with sets of base features identified asimplementations.

Other implementations may include a non-transitory computer readablestorage medium storing instructions executable by a processor to performany of the methods described above. Yet another implementation mayinclude a system including memory and one or more processors operable toexecute instructions, stored in the memory, to perform any of themethods described above.

FIG. 10 illustrates one implementation of a method 1000 of determiningintent for an interaction between a control object in athree-dimensional (3D) sensory space and a movable virtual object in avirtual space that the control object interacts with. At action 1002, aplurality of virtual feeler zones of the control object are defined. Inone implementation, the virtual feeler zones reproduce movements of thecontrol object in a virtual space that are captured in the 3D sensoryspace. In another implementation, a synthetic or augmented space iscreated by overlaying the virtual space on a physical space and thevirtual feeler zones reproduce movements of the control object in thesynthetic or augmented space.

At action 1012, a plurality of forces are applied to the movable virtualobject at one or more points of virtual contacts of the feeler zoneswith the virtual object in response to detecting a controlobject-gesture. At action 1022, a center of effort for the forcesapplied by the feeler zones is calculated by determining a geometriccentroid of a polygon forced by the points of virtual contacts.

At action 1032, a gesture-type is determined based on movement of thepoints of virtual contacts and the center of effort. In oneimplementation, the gesture-type is identified as a translational strokein response to detecting displacement of the center of effort. Inanother implementation, the gesture-type is identified as a rotationalstroke in response to detecting rotation of the points of virtualcontacts around the center of effort.

At action 1042, an unrealistic gesture-type is recognized when adistance between the feeler zones exceeds a realistic threshold. In oneimplementation, the realistic threshold can be determined from referencesources or experimentally measured data. In another implementation, analternative gesture-type is selected based on an unrealistic position ofa particular feeler zone that exceeded the realistic threshold.

At action 1052, the virtual object is manipulated responsive to thedetermined gesture-type. In one implementation, the manipulationincludes shaking, moving, or rotating the virtual object. In anotherimplementation, manipulation includes selecting, grabbing, or liftingthe virtual object.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationsmay include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation may include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 11 is a flowchart showing a method 1100 of determining a degree ofprecision of a gesture in accordance with implementations of thetechnology disclosed. At action 1102, a plurality of virtual feelerzones of the control object are defined. In one implementation, thevirtual feeler zones reproduce movements of the control object in avirtual space that are captured in the 3D sensory space. In anotherimplementation, a synthetic or augmented space is created by overlayingthe virtual space on a physical space and the virtual feeler zonesreproduce movements of the control object in the synthetic or augmentedspace.

At action 1112, a plurality of forces are applied to the movable virtualobject at one or more points of virtual contacts of the feeler zoneswith the virtual object in response to detecting a controlobject-gesture. At action 1122, a degree of precision of the controlobject-gesture is determined responsive to a number of points of virtualcontacts of the feeler zones and proximities between the points ofvirtual contacts. In one implementation, the control object-gesture isdetermined to be a gross motor gesture when the degree of precisionexceeds a precision threshold value. In another implementation, thecontrol object-gesture is determined to be a fine motor gesture when thedegree of precision deceeds a precision threshold value.

At action 1132, the virtual object is manipulated based on thedetermined degree of precision. In one implementation, the manipulationincludes shaking, moving, or rotating the virtual object. In anotherimplementation, manipulation includes selecting, grabbing, or liftingthe virtual object.

Other implementations can include a non-transitory computer readablestorage medium storing instructions executable by a processor to performany of the methods described above. Yet another implementation caninclude a system including memory and one or more processors operable toexecute instructions, stored in the memory, to perform any of themethods described above.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationsmay include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation may include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

The terms and expressions employed herein are used as terms andexpressions of description and not of limitation, and there is nointention, in the use of such terms and expressions, of excluding anyequivalents of the features shown and described or portions thereof. Inaddition, having described certain implementations of the technologydisclosed, it will be apparent to those of ordinary skill in the artthat other implementations incorporating the concepts disclosed hereincan be used without departing from the spirit and scope of thetechnology disclosed. Accordingly, the described implementations are tobe considered in all respects as only illustrative and not restrictive.

What is claimed is:
 1. A method of determining intent for an interactionbetween a hand in a three-dimensional (3D) sensory space and a movablevirtual object in a virtual space that the hand interacts with, themethod including: defining at distal ends of fingers of a hand thatserves as a control object, a plurality of feeler zones for display uponestablishing one or more points of virtual contacts between the distalends of fingers of the hand and corresponding locations anywhere on anysurface of a virtual object, each feeler zone, of the feeler zones,representing a virtual location in the virtual space that corresponds toa location tangent to a particular finger of the hand in the 3D sensoryspace, as tracked by a camera based sensor; and responsive to detectinga hand-gesture made by fingers or the hand, manipulating the virtualobject according to forces applied to the movable virtual object at theone or more points of virtual contacts established at the feeler zonesat the distal ends of fingers of a hand with the corresponding locationsanywhere on any surface of the virtual object.
 2. The method of claim 1,further including: identifying a gesture-type for the hand-gesture madeby the fingers or the hand as a translational stroke in response todetecting displacement of a center of effort obtained for the forcesapplied by the feeler zones.
 3. The method of claim 1, furtherincluding: identifying a gesture-type for the hand-gesture made by thefingers or the hand as a rotational stroke in response to detectingrotation of the one or more points of virtual contacts around the centerof effort obtained for the forces applied by the feeler zones.
 4. Themethod of claim 1, further including: identifying a gesture-type for thehand-gesture made by the fingers or the hand as a trans-rotationalstroke in response to detecting rotation of the one or more points ofvirtual contacts around the center of effort and displacement of thecenter of effort obtained for the forces applied by the feeler zones. 5.The method of claim 1, further including recognizing an unrealisticgesture-type when a distance between the feeler zones exceeds arealistic threshold.
 6. The method of claim 5, further includingselecting an alternative gesture-type based on an unrealistic positionof a particular feeler zone that exceeded the realistic threshold. 7.The method of claim 1, further including: determining a degree ofprecision of the hand-gesture responsive to at least one selected from aset of points of virtual contacts of the feeler zones and proximitiesbetween select ones of points of virtual contacts; and manipulating thevirtual object according to the forces applied by the feeler zones atthe distal ends of fingers of a hand that serves as a control objectbased on the hand-gesture with the degree of precision determined. 8.The method of claim 7, further including: determining the hand-gestureto be a gross motor gesture when the degree of precision exceeds aprecision threshold value.
 9. The method of claim 7, further including:determining the hand-gesture to be a fine motor gesture when the degreeof precision deceeds a precision threshold value.
 10. A system fordetermining intent for an interaction between a hand in athree-dimensional (3D) sensory space and a movable virtual object in avirtual space that the hand interacts with, the system including: acamera based sensor; and a processor and a computer readable storagemedium storing computer instructions configured to cause the processorto: define at distal ends of fingers of a hand that serves as a controlobject, a plurality of feeler zones for display upon establishing one ormore points of virtual contacts between the distal ends of fingers ofthe hand and corresponding locations anywhere on any surface of avirtual object, each feeler zone, of the feeler zones, representing avirtual location in the virtual space that corresponds to a locationtangent to a particular finger of the hand in the 3D sensory space, astracked by the camera based sensor; and responsive to detecting ahand-gesture made by fingers or the hand, manipulate the virtual objectaccording to forces applied to the movable virtual object at the one ormore points of virtual contacts established at the feeler zones at thedistal ends of fingers of a hand with the corresponding locationsanywhere on any surface of the virtual object.
 11. The system of claim10, further configured to: identify a gesture-type for the hand-gesturemade by the fingers or the hand as a translational stroke in response todetecting displacement of the center of effort obtained for the forcesapplied by the feeler zones.
 12. The system of claim 10, furtherconfigured to: identify a gesture-type for the hand-gesture made by thefingers or the hand as a rotational stroke in response to detectingrotation of the one or more points of virtual contacts around the centerof effort obtained for the forces applied by the feeler zones.
 13. Thesystem of claim 10, further configured to: identify a gesture-type forthe hand-gesture made by the fingers or the hand as a trans-rotationalstroke in response to detecting rotation of the one or more points ofvirtual contacts around the center of effort and displacement of thecenter of effort obtained for the forces applied by the feeler zones.14. The system of claim 10, further configured to recognize anunrealistic gesture-type when a distance between the feeler zonesexceeds a realistic threshold.
 15. The system of claim 14, furtherconfigured to select an alternative gesture-type based on an unrealisticposition of a particular feeler zone that exceeded the realisticthreshold.
 16. The system of claim 10, further including: the computerreadable storage medium storing computer instructions configured tocause the processor to: determine a degree of precision of thehand-gesture responsive to at least one selected from a set of a numberof points of virtual contacts of the feeler zones and proximitiesbetween select ones of points of virtual contacts; and manipulate thevirtual object according to the forces applied by the feeler zones atdistal ends of fingers of a hand that serves as a control object, basedon the hand-gesture with degree of precision determined.
 17. The systemof claim 16, further configured to: determine the hand-gesture to be agross motor gesture when the degree of precision exceeds a precisionthreshold value.
 18. The system of claim 16, further configured to:determine the hand-gesture to be a fine motor gesture when the degree ofprecision deceeds a precision threshold value.
 19. One or morenon-transitory computer readable media having instructions storedthereon for determining intent for an interaction between a hand in athree-dimensional (3D) sensory space and a movable virtual object in avirtual space that the hand interacts with, which instructions whenexecuted by one or more processors perform actions including: definingat distal ends of fingers of a hand that serves as a control object, aplurality of feeler zones for display upon establishing one or morepoints of virtual contacts between the distal ends of fingers of thehand and corresponding locations anywhere on any surface of a virtualobject, each feeler zone, of the feeler zones, representing a virtuallocation in the virtual space that corresponds to a location tangent toa particular finger of the hand in the 3D sensory space, as tracked by acamera based sensor; and responsive to detecting a hand-gesture made byfingers or the hand, manipulating the virtual object according to forcesapplied to the movable virtual object at the one or more points ofvirtual contacts established at the feeler zones at the distal ends offingers of a hand with the corresponding locations anywhere on anysurface of the virtual object.
 20. The one or more non-transitorycomputer readable media of claim 19, further having instructions storedthereon for performing: determining a degree of precision of thehand-gesture responsive to at least one selected from a set of a numberof points of virtual contacts of the feeler zones and proximitiesbetween select ones of points of virtual contacts; and manipulating thevirtual object according to the forces applied by the feeler zones atdistal ends of fingers of a hand that serves as a control object, basedon the hand-gesture with the degree of precision determined.